
NCVER 



Pi 

W 

PLh 

< 

Ph 

w 

u 

£ 

w 

w 

rT 

£ 

o 

u 



Measuring educational outcomes: 
Vocational education and training 

TOM KARMEL 

National Centre for Vocational 
education Research 



Paper presented to 

NatStats08 conference W'orking together 

FOR AN INFORMED AUSTRALIAN SOCIETY, 

19-21 November 2008, Melbourne 



The views and opinions expressed in this document are those of 
the author and do not necessarily reflect the views of the 
Australian Government or state and territory governments. 



© National Centre for Vocational Education Research, 2009 

This work has been produced and published by the National Centre for Vocational Education Research 
(NCVER). Apart from any use permitted under the Copyright Act 1968 , no part of this publication may be 
reproduced by any process without written permission. Requests should be made to NCVER. 

The National Centre for Vocational Education Research is an independent body responsible for collecting, 
managing and analysing, evaluating and communicating research and statistics about vocational education and 
training (VET). 

NCVER’s inhouse research and evaluation program undertakes projects which are strategic to the VET 
sector. These projects are developed and conducted by NCVER’s research staff and are funded by NCVER. 
This research aims to improve policy and practice in the VET sector. 



TD/TNC 95.30 

Published by NCVER 
ABN 87 007 967 311 

Level 1 1, 33 King William Street, Adelaide, SA 5000 
PO Box 8288 Station Arcade, Adelaide SA 5000, Australia 

ph +61 8 8230 8400 fax +61 8 8212 3436 
email ncver@ncver.edu.au 
<http:/ / www.ncver.edu.au> 

<http://www.ncver.edu.au/ publications/ 2128.html> 




About the research 




NCVER 



Measuring educational outcomes: b Vocational education and training 

Tom Karmel, NCVER 



This paper was presented to the NatStats08 conference Working together for an informed Australian 
society in November 2008. 

The vocational education and training (VET) sector has a long tradition of measuring and reporting 
outcomes. The public face of this is the Annual national report of the Australian vocational education and 
training system published (and tabled in the Commonwealth Parliament) since 1994. The reporting 
framework has undergone a number of changes corresponding to revision in high-level strategies 
developed by the former Australian National Training Authority (ANTA). 1 This, however, is about 
to change. The catalyst for a radical examination of the reporting framework is the Council of 
Australian Governments’ (COAG) reform agenda. This reform agenda is shaking up the funding 
relationships between the Commonwealth and the states and will place increasing reliance on 
measuring and reporting outcomes as distinct from focusing on the resources used (that is, inputs). 

In this paper, I provide a history of performance measurement for the VET sector, beginning with 
the creation of the Australian National Training Authority and ending with what we know of the 
current reforms. As well as describing the various measures, I discuss the challenges that are 
thrown up by indicators. I conclude with my suggestions for indicators for the vocational education 
and training system. 



1 In 2005 the Australian National Training Authority was abolished and its functions assumed by the Department of 
Education, Science and Training (now the Department of Education, Employment and Workplace Relations). 
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History 



The Australian National Training Authority was established in 1992 by the Australian National 
Training Authority Act 1992 (ANTA Act). Under ANTA, three national strategies were developed to 
provide direction and a performance-monitoring framework. 

The first national strategy for vocational education and training, Towards a skilled Australia , was 
released in 1994 to cover the period 1994—98. 

The strategy was developed around four themes: 

Objective l.l 2 : Responsiveness, so that diversity, choice and cooperation was maximised 
between the full range of training providers — public, private and industry 

Objective 1.2: Quality, so that those achieving at the highest standards are supported and 

incentives are offered to others to reach those standards 

Objective 1.3: Accessibility, so that all Australians who want and need training can get it 

Objective 1.4: Efficiency, so that value for money and accountability are emphasised and 

administrative arrangements are streamlined and simplified. 

The performance measures were: 

KPM 1.1 Actual versus target student load 

KPM 1.2 Module load completion rates 

KPM 1.3 Training completion numbers 

KPM 1.4 Average cost per student contact hour. 

The second strategic report to provide direction for vocational education in Australia was A bridge 
to the future (1998—2003). 

The five key objectives of this strategy were: 

Objective 2.1: Enhancing mobility in the labour market 

Objective 2.2: Equipping Australians for the world of work 

Objective 2.3: Achieving equitable outcomes in VET 

Objective 2.4: Maximising the value of public VET expenditure 

Objective 2.5: Increasing investment in training. 



2 In order to make it easier to keep track of the objectives and indicators as they changed with new versions of the 
national strategy, we adopt the convention of the first digit representing which strategy. So objective 1.1 is the first 
objective of the first national strategy. 



6 



Measuring educational outcomes: Vocational education and training 




This strategy had an expanded number of indicators: 

KPM 2.1: Skill outputs produced annually within the domain of formally recognised 

vocational education and training 

KPM 2.2: Stocks of vocational education and training skills against desired levels 

KPM 2.3: Employers’ views on the relevance of skills acquired through vocational education 

and training 

KPM 2.4: Student employment outcomes and prospects before and after participation in 

vocational education and training 

KPM 2.5: VET client groups’ participation, outputs and outcomes 

KPM 2.6: Public expenditure per publicly funded output 

KPM 2.7: Public expenditure per total recognised output 

KPM 2.8: Total expenditure on vocational education and training. 

Although each key performance measure looks at only one particular performance aspect, when 
considered together, they were intended to provide a comprehensive picture of the outputs, 
efficiency and effectiveness of the vocational education and training system. 

In regards to the third objective, there were also five identified equity groups for which close 
monitoring of participation, outputs and outcomes occurred. These were women, Indigenous 
people, people of non-English speaking background, people with a disability and people from rural 
or remote areas. 

The final national strategy created by ANTA, prior to its demise, was Shaping our future, which had 
the following objectives: 

Objective 3.1: Industry will have a highly skilled workforce to support strong performance in 

the global economy. 

Objective 3.2: Employers and individuals will be at the centre of vocational education and 

training. 

Objective 3.3: Communities and regions will be strengthened economically and socially 

through learning and employment. 

Objective 3.4: Indigenous Australians will have skills for viable jobs and their learning culture 

will be shared. 

Six key performance measures (KPMs) were created: 

KPM 3.1: Student participation and achievement in vocational education and training 

KPM 3.2: Student employment outcomes and satisfaction with VET 

KPM 3.3: Employer engagement and satisfaction with VET 

KPM 3.4: VET outcomes for Indigenous Australians 

KPM 3.5: Community engagement and satisfaction with VET 

KPM 3.6: VET system efficiency. 
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What is very noticeable about the three national strategies is how few of the objectives have lasted 
the full period. As can be seen from box 1, only the equity objective is common to all three 
strategies. Three of the themes — efficiency, quality of training and workforce quality — are common 
to two out of the three strategies. 3 The remaining themes — community strength, being client 
driven, responsiveness — featured in only one of the three strategies. 



Box 1 Summary of objectives 



Objectives 


Strategy 




Equity 


1 


Access for all Australians 




2 


Achieve equitable outcomes in VET 




3 


Indigenous Australians having skills for viable jobs and a shared learning 
culture 


Efficiency 


1 


Value for money, accountability, administrative arrangements 
streamlined 




2 


Maximising the value of public VET expenditure 


Quality of training 


1 


Support high achievers/incentives to reach standards 




2 


Increase investment in training 


Workforce quality 


2 


Enhance mobility in the labour market 




2 


Equip Australians for the world of work 




3 


Highly skilled workforce to support global economy 


Community strength 


3 


Strengthen economically and socially 


Client-driven 


3 


Employers and individuals at the centre of VET 


Responsiveness of 
providers 


1 


Diversity of choice and cooperation between public, private and industry 
training providers 



The first very noticeable thing about the indicators is that they do not map particularly well to the 
objectives. Box 2 categorises the indicators according to four headings: efficiency, inputs, outputs 
and outcomes. These headings are common to indicator systems and tend to suggest that the 
indicators were developed by people with experience in indicators rather than coming from the 
objectives themselves. Indeed, one could be forgiven for thinking that the indicators were 
developed largely independently of the objectives. The fifth heading — equity — cuts across the 
inputs, outputs and outcomes and is probably best thought of as a particular cut of the other 
categories. Another comment is that some of the objectives appeared to be elusive (in particular, 
being client-driven and providers being responsive) and none of the indicators appears to relate to 
them. Finally, it should be noted that the indicator for community strength — KPM 3.5: Community 
engagement and satisfaction with VET — was abandoned because it proved to be impossible to 
collect. 



3 Including the quality of training theme is arguable. Is ‘increasing investment in training’ about quality or quantity? 
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Box 2 



A classification of the indicators 



Efficiency measures 


KPM 


1.1 




KPM 


1.2 




KPM 


1.4 




KPM 


2.6 




KPM 


2.7 




KPM 


3.6 


Inputs (participation in training) 


KPM 


2.8 




KPM 


3.1 




KPM 


2.5 


Outputs (skills acquired) 


KPM 


1.3 




KPM 


2.1 




KPM 


2.5 




KPM 


3.1 


Outcomes (meeting the needs of the 


KPM 


2.2: 


economy and the community) 








KPM 


2.3 




KPM 


2.4 




KPM 


2.5 




KPM 


3.2 




KPM 


3.3 




KPM 


3.4 




KPM 


3.5 


Equity indicators 


KPM 


2.5 




KPM 


3.4 



Actual versus target student load 
Module load completion rates 
Average cost per student contact hour 
Public expenditure per publicly funded output 
Public expenditure per total recognised output 
VET system efficiency 

Total expenditure on vocational education and training 

Student participation and achievement in vocational 
education and training 

VET client groups' participation (outputs and outcomes) 
Training completion numbers 
Skill outputs 

VET client groups (participation), outputs (and outcomes) 

(Student participation) and achievement in vocational 
education and training 

Stocks of vocational education and training skills against 
desired levels 

Employers’ views on the relevance of skills 
Student employment outcomes before and after 
VET client groups (participation, outputs) and outcomes 
Student employment outcomes and satisfaction with VET 
Employer engagement and satisfaction with VET 
VET outcomes for Indigenous Australians 
Community engagement and satisfaction with VET 
VET client groups’ participation (outputs and outcomes) 
VET outcomes for Indigenous Australians 
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Current developments 



The Council of Australian Governments (COAG) created the working group on the productivity 
agenda (PAWG), which promises to rewrite the objectives of the training system and associated 
indicators. While PAWG also covered issues related to early childhood and school education, our 
interest is the objectives developed for the training system (skills and development). Four goals 
have been set: 

Objective 4.1: That the working-age population has gaps in foundation skills levels reduced to 

enable effective educational, labour market and social participation. 

Objective 4.2: The working-age population has the depth and breadth of skills and capabilities 

required for the twenty-first century labour market. 

Objective 4.3: The supply of skulls provided by the national training system responds to meet 

changing labour market demand. 

Objective 4.4: Skills are used effectively to increase labour market efficiency, productivity, 

innovation, and ensure increased utilisation of human capital. 

The following KPMs have been chosen to measure progress against these goals. 

KPM 4.1: Proportion of the working- age population at literacy level one, two or three. 

KPM 4.2: Proportion of 20 to 64-year-olds who do not have qualifications at or above a 

certificate III. 

KPM 4.3: Proportion of graduates employed after completing training, by previous 

employment status. 

KPM 4.4: The percentage of graduates with improved employment status after training. 

KPM 4.5: The number of hard-to-fill vacancies. 

KPM 4.6: Proportion of people employed at or above the level of their qualification, by field 

of study. 

These KPMs map closely to the four objectives: KPM 4.1 links to Objective 4.1; KPM 4.2 links to 
Objective 4.2; KPM 4.3 and 4.4 link to Objective 4.3, and the final KPMs link to Objective 4.3. 
What is interesting — by comparison with the indicators for the earlier national strategies — is that 
there are no indicators of efficiency, and no indicators of inputs. All the measures focus on outputs 
and outcomes, and the output measures do not, as is more conventional, look at what the system is 
producing direcdy. Rather, they focus on characteristics of the overall population. 4 

This approach is a radical departure from the earlier approaches. Whether the indicators are up to 
the task that has been set will be a matter for the future. The concepts embodied in the four 



4 The indicators do not explicitly mention equity. However, there is an understanding that the indicators will be cut by 
various sub-groups, although this may stretch the available data in some cases. 
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objectives are complex and it is difficult to come up with a small number of easy-to-compile 
indicators that will do justice to the objectives. 5 



Some ruminations on indicators 

The development of indicators is difficult. Often objectives are difficult to define in a way that 
invites measurement. Data are rarely good enough to support precise measurement. Indicators 
must be simple to explain. In developing a set of indicators I point to the following issues. 

The first issue is clarity of objectives. As can be seen from the earlier discussion in which objectives 
changed significantly over a relatively short period, the objectives for the VET sector are by no 
means obvious. Compared with the health sector, for example, objectives can be contested. Is the 
objective of the VET sector to produce qualified workers? Or should it be about improving the 
productivity of the workforce? Or is a key objective to be flexible and respond to client demands? 
Compare this with the health sector, where most would agree that the health sector should be 
concerned with improving the health of the population. 

The second issue relates to indicators. I list a series of relevant attributes. 

Relevance-, do the indicators relate to objectives? It is always tempting to measure what is measurable. 

Tractability : indicators must be able to be compiled. For example, KPM 3.5 proved to be totally 
intractable and was abandoned. 

Confoundingfactors-. indicators need to relate to objectives, but often there are confounding factors. 
For example, the number of skilled vacancies is a perfectly reasonable measure of the extent to 
which the training system is meeting the needs of the labour market. However, there are 
confounding factors such as the state of the economy and inadequate working conditions and 
wages. 

Disaggregation issues-, policy may dictate that special attention is given to sub-groups, but the data may 
not be up to the task. An obvious case here is indicators for Indigenous people, where 
measurement issues are very difficult. 

Unintended consequences-, if resources are allocated on the basis of indicators, then there is the risk of 
perverse behaviour. For example, if high completion rates are rewarded, then providers may have 
an incentive to lower standards or to exclude those who are more difficult to teach. 

Avoiding the hard issues-, in VET, collecting data from private providers on private training has not yet 
been achieved, and hence they have been excluded from indicators to date. Similarly, relative wage 
rates are relevant to measuring the value of qualifications, but they are more politically difficult 
because there is a reluctance to say that qualifications are not valuable even though they are not 
rewarded in the labour market. Completion rates would appear to be important and these have 
proved difficult to calculate. 

Robustness-, if indicators are used to allocate resources, then the indicators must be robust so that 
unfair allocations are not made on the basis of poor-quality measures. In this regard, it should be 
remembered that indicators ‘indicate’ and labelling them ‘measures’, as in KPMs, provides the 
potential to give them a status they do not deserve. 



5 I need to declare a conflict of interest here. NCVER was asked to provide advice on a number of proposed indicators. 
In that advice a rather larger number of indicators were proposed, some of much greater complexity. The working 
group selection concentrated on indicators that are relatively straightforward. 
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Indicators are not a substitute for analysis-, the world is a complex place and it is naive to believe that 
policy outcomes can be adequately assessed through simple indicators. 

Simplicity is not always a virtue-, policy-makers tend to like simple indicators. However, simplicity is not 
always a help to understanding. For example, indicators are often calculated for the whole 
population, but age-specific measures can be more useful. Indicators at a whole-of-population level 
are influenced by changes in demographic shares, and therefore can be misleading. 



KarmePs indicators 

We have seen that the objectives and the indicators for VET have been many and varied over the 
last 15 or so years. I thought it would be interesting to start with a clean sheet of paper and see 
what I come up with and then see how this compares with what we have seen. 

First, I need to define the objectives. I would have two: a skilled workforce and an equitable 
training system. By a skilled workforce I mean a workforce that is productive; a more skilled 
workforce means a more productive workforce, not a more qualified workforce. By an equitable 
training system I mean one which provides opportunities and good outcomes for specified groups 
of the population. That is, a ‘socially inclusive’ system which benefits all. 

I would have indicators under three groupings: outcomes, equity and efficiency. 



Outcomes 

If we think of VET as adding to human capital, then we would want to know the rate at which 
Australia’s human capital is increasing. Indicators about the proportion of people with 
qualifications are an obvious measure. However, such a measure is very partial. In particular, 
qualifications are of no use if they do not attract a return. So I would be looking for evidence that 
vocational qualifications are valued in the workforce. The measures of this are employment rates 
and wage rates. With some trouble these can be combined into a measure of workforce quality 
(akin to the Australian Bureau of Statistics [ABS] measures of factor productivity). 

Thus my indicators would be: 

^ qualification levels in the working- age population 
^ relative employment rates (by qualification type) 

^ relative wages (by qualification type) 

^ overall quality of the workforce. 

These are stock measures and will move fairly slowly. So it would be sensible to supplement them 
with measures associated with the output of the training system. What I have in mind here are 
indicators such as: 

^ proportion of graduates 6 who improve their employment position (for example, not employed 
to employed) 

^ proportion of graduates who report that their training is relevant to their current employment. 

In terms of equity, I would be looking at three aspects: participation, qualification levels, and return 
to those qualifications. The first two of these are perhaps necessary conditions, but they 



6 We would also differentiate between those who have completed their qualification (VET graduates) and those who 
have not completed a full qualification (module completers). 
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are not sufficient for a socially inclusive training system. Equity will only be satisfied if the group in 
question is reaping the rewards of participation. So my indicators would be: 

^ proportion of target population participating in VET 

^ numbers of graduates from the target population 

^ employment outcomes from graduates (and others who have completed their training but not 
completed a qualification). These employment outcomes would be compared with outcomes 
from the target population but without training, and with outcomes from similar graduates from 
the wider population. 



Efficiency 

Efficiency is a difficult concept. While the ratio of outputs to inputs is an obvious measure, it does 
not automatically include a quality dimension. Quality is difficult to measure, but student 
satisfaction provides a plausible indication. Another obvious measure is the completion rate. 
However, such a rate is of more relevance to some groups rather than to others. For example, we 
know that outcomes are generally better for people who are upgrading their qualifications, but not 
necessarily for those who are broadening their skills. Thus completion rates of new entrants to the 
labour market are most likely to be of more importance than completion rates of older people, who 
may or may not be upgrading qualifications. 

So my indicators of efficiency would be: 

^ unit cost of provision of training per completed module/ unit 
^ student satisfaction 

^ qualification completion rates for people up to 24 years 
^ qualification completion rates for those with no post-school qualification. 

My suggestions have some similarities with indicators used to date. Employment outcomes, student 
satisfaction, and unit cost measures have all been published for many years. The main difference in 
my selection is the use of relative wages and relative employment rates, and the lack of participation 
rates (apart from equity groups). The latter is also a feature of the indicators proposed by the 
working group on the productivity agenda (PAWG), which relate primarily to the population rather 
than participation in training. 

My emphasis is driven by a view that there is an optimal amount of training and that too much 
training can be a poor use of resources. Thus I am looking for indicators that reflect improvements 
in the quality of the workforce which go further than measuring numbers of people with 
qualifications. The supply of people with skills must be appropriate for labour demand rather than 
an end in its own right. The PAWG indicator KPM 4.6 relating to appropriate skill usage is also 
trying to address this. 7 1 also have not included an indicator to measure the VET system’s 
responsiveness (which is the aim of KPM 4.5: hard-to-fill vacancies). This is because such 
indicators tend to be confounded by economic conditions and therefore give little evidence on the 
VET sector as such. This is an example of where more sophisticated analysis is needed rather than 
simple indicators. 



7 I did not include this indicator because overskilling is a complex issue. One problem is that there are significant 
boundary effects. For example, an individual with a very low-level qualification, by definition, will be working at that 
level or higher. Similarly, an individual with a very high-level qualification will typically be working at that level or 
lower. 
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A final comment 



My personal view is that indicators are most useful when they aid understanding. In most cases, 
indicators ‘indicate’ rather than measure precisely. They should provide sufficient information to 
provoke questions, and we should not be too concerned when they fail to provide easy answers. If 
this argument is accepted, then governments should be wary of tying resource allocations to simple 
indicators. Rarely are they robust enough to be the basis of funding decisions. In the same vein, 
indicators should only be a start; they do not take the place of detailed and thoughtful analysis. 
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